Image processing device, image processing method, image processing program, and diagnosis support system

ABSTRACT

An image processing device 100 includes, in a case where designation of a plurality of partial regions corresponding to a cell morphology is received, the plurality of partial regions being extracted from a pathological image, a generation unit 154 that generates auxiliary information indicating information about a feature amount effective when a plurality of partial regions is classified or extracted with respect to a plurality of feature amounts calculated from the image; and in a case where setting information about an adjustment item according to the auxiliary information is received, an image processing unit 155 that performs an image process on the image using the setting information.

FIELD

The present invention relates to an image processing device, an imageprocessing method, an image processing program, and a diagnosis supportsystem.

BACKGROUND

There is a system that photographs an observation target placed on aglass slide with a microscope, generates a digitized pathological image,and performs various types of image analyses on the pathological image.For example, the observation target is a tissue or a cell collected froma patient, and corresponds to a piece of meat of an organ, saliva,blood, or the like.

As a conventional technique related to image analysis, a technique ofinputting a pathological image to a morphology detector, detecting amorphology or a state of a cell nucleus, a cell membrane, or the likeincluded in the pathological image, and calculating a feature amountobtained by quantifying a feature of the morphology is known. A skilledperson such as a pathologist or a researcher sets adjustment items of anidentifier for classifying or extracting a morphology or a state havinga specific feature based on the calculation result of the feature amountand the specialized knowledge.

CITATION LIST Patent Literature

Patent Literature 1: JP 2018-502279 W

SUMMARY Technical Problem

It is difficult for a user with poor specialized knowledge to associatea feature amount obtained by quantifying a feature of a morphology and astate calculated by the prior art with a feature based on thespecialized knowledge of the user, and there is room for improvement.

Therefore, the present disclosure proposes an image processing device,an image processing method, an image processing program, and a diagnosissupport system capable of appropriately displaying a feature amountobtained by quantifying an appearance feature of a morphology and easilysetting an adjustment item of an identifier.

Solution to Problem

To solve the problems described above, an image processing deviceaccording to an embodiment of the present disclosure includes: in a casewhere designation of a plurality of partial regions corresponding to acell morphology is received, the plurality of partial regions beingextracted from a pathological image, a generation unit that generatesauxiliary information indicating information about a feature amounteffective when a plurality of partial regions is classified or extractedwith respect to a plurality of feature amounts calculated from theimage; and in a case where setting information about an adjustment itemaccording to the auxiliary information is received, an image processingunit that performs an image process on the image using the settinginformation.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a diagnosis support system according tothe present embodiment.

FIG. 2 is a diagram for explaining an imaging process according to thepresent embodiment.

FIG. 3 is a diagram for explaining an imaging process according to thepresent embodiment.

FIG. 4 is a diagram for explaining a process of generating a partialimage (tile image).

FIG. 5 is a diagram for explaining a pathological image according to thepresent embodiment.

FIG. 6 is a diagram for explaining a pathological image according to thepresent embodiment.

FIG. 7 is a diagram illustrating an example of a browsing mode by aviewer of a pathological image.

FIG. 8 is a diagram illustrating an example of a browsing historystorage unit included in a server.

FIG. 9A is a diagram illustrating a diagnostic information storage unitincluded in a medical information system.

FIG. 9B is a diagram illustrating a diagnostic information storage unitincluded in the medical information system.

FIG. 9C is a diagram illustrating a diagnostic information storage unitincluded in the medical information system.

FIG. 10 is a diagram illustrating an example of an image processingdevice according to the present embodiment.

FIG. 11 is a diagram illustrating an example of a data structure of apathological image DB.

FIG. 12 is a diagram illustrating an example of a data structure of afeature amount table.

FIG. 13 is a diagram illustrating an example of a partial regionextracted from a pathological image.

FIG. 14 is a diagram for explaining a process by a display control unit.

FIG. 15 is a diagram illustrating an example of first auxiliaryinformation.

FIG. 16 is a diagram illustrating an example of second auxiliaryinformation.

FIG. 17 is a diagram illustrating another display example of the secondauxiliary information.

FIG. 18 is a diagram illustrating an example of third auxiliaryinformation.

FIG. 19 is a diagram illustrating an example of fourth auxiliaryinformation.

FIG. 20 is a diagram for explaining an example of a classificationprocess performed by an image processing unit.

FIG. 21 is a diagram for explaining an example of a classificationprocess performed by an image processing unit.

FIG. 22 is a diagram for explaining an example of a classificationprocess performed by an image processing unit.

FIG. 23 is a flowchart illustrating a processing procedure of the imageprocessing device according to the present embodiment.

FIG. 24 is a diagram for explaining another process by the imageprocessing device.

FIG. 25 is a diagram for explaining another process by the imageprocessing device.

FIG. 26 is a hardware configuration diagram illustrating an example of acomputer that implements functions of an image processing device.

DESCRIPTION OF EMBODIMENTS

Hereinafter, the embodiments of the present disclosure will be describedin detail with reference to the drawings. In the following embodiments,the same parts are denoted by the same reference signs, and a duplicatedescription will be omitted.

Further, the present disclosure will be described in the order of thefollowing items.

<Present Embodiment>

1. Configuration of system according to present embodiment

2. Various kinds of information

2-1. Pathological image

2-2. Browsing history information

2-3. Diagnostic information

3. Image processing device according to present embodiment

4. Processing procedure

5. Another process

6. Effects of image processing device according to present embodiment

7. Hardware configuration

8. Conclusion

Present Embodiment 1. Configuration of System According to PresentEmbodiment

First, a diagnosis support system 1 according to the present embodimentwill be described with reference to FIG. 1 . FIG. 1 is a diagramillustrating a diagnosis support system 1 according to the presentembodiment. As illustrated in FIG. 1 , the diagnosis support system 1includes a pathology system 10 and an image processing device 100.

The pathology system 10 is a system mainly used by a pathologist, and isapplied to, for example, a laboratory or a hospital. As illustrated inFIG. 1 , the pathology system 10 includes a microscope 11, a server 12,a display control device 13, and a display device 14.

The microscope 11 is an imaging device that has a function of an opticalmicroscope, captures an image of an observation target placed on a glassslide, and acquires a pathological image that is a digital image. Notethat the observation target is, for example, a tissue or a cellcollected from a patient, and is a piece of meat of an organ, saliva,blood, or the like.

The server 12 is a device that stores and holds a pathological imagecaptured by the microscope 11 in a storage unit (not illustrated). In acase of receiving the browsing request from the display control device13, the server 12 searches for a pathological image from a storage unit(not illustrated) and transmits the searched pathological image to thedisplay control device 13. In addition, in a case of receiving theacquisition request for the pathological image from the image processingdevice 100, the server 12 searches for the pathological image from thestorage unit and transmits the searched pathological image to the imageprocessing device 100.

The display control device 13 transmits a browsing request for thepathological image received from the user to the server 12. Then, thedisplay control device 13 causes the display device 14 to display thepathological image received from the server 12.

The display device 14 has a screen using, for example, liquid crystal,electro-luminescence (EL), cathode ray tube (CRT), or the like. Thedisplay device 14 may be compatible with 4K or 8K, or may be formed by aplurality of display devices. The display device 14 displays thepathological image controlled by the display control device 13. Notethat, although details will be described later, the server 12 stores thebrowsing history information about the region of the pathological imageobserved by the pathologist via the display device 14.

The image processing device 100 is a device that transmits anacquisition request for a pathological image to the server 12 andperforms the image process on the pathological image received from theserver 12.

2. Various Kinds of Information

[2-1. Pathological Image]

As described above, the pathological image is generated by imaging theobservation target with the microscope 11. First, an imaging process bythe microscope 11 will be described with reference to FIGS. 2 and 3 .FIGS. 2 and 3 are diagrams for explaining the imaging process accordingto the first embodiment. The microscope 11 described below includes alow-resolution imaging unit for performing imaging at low resolution anda high-resolution imaging unit for performing imaging at highresolution.

In FIG. 2 , a glass slide G10 on which an observation target A10 isplaced is included in an imaging region R10 that is a region that can beimaged by the microscope 11. The glass slide G10 is placed on a stage(not illustrated), for example. The microscope 11 images the imagingregion R10 by the low-resolution imaging unit to generate an entireimage that is a pathological image in which the observation target A10is entirely imaged. In label information L10 illustrated in FIG. 2 ,identification information (for example, a character string or a QR code(registered trademark)) for identifying the observation target A10 isdescribed. By associating the identification information described inthe label information L10 with the patient, it is possible to identifythe patient corresponding to the entire image. In the example of FIG. 2, “#001” is described as the identification information. In the labelinformation L10, for example, a simple description of the observationtarget A10 may be described.

Subsequently, after generating the entire image, the microscope 11identifies the region where the observation target A10 exists from theentire image, and sequentially images, by the high-resolution imagingunit, each divided region obtained by dividing the region where theobservation target A10 exists for each predetermined size. For example,as illustrated in FIG. 3 , the microscope 11 first images a region R11to generate a high-resolution image I11 that is an image illustrating apartial region of the observation target A10. Subsequently, themicroscope 11 moves the stage to image a region R12 by thehigh-resolution imaging unit to generate a high-resolution image I12corresponding to the region R12. Similarly, the microscope 11 generateshigh-resolution images I13, I14, . . . , corresponding to regions R13,R14, . . . , respectively. Although regions up to the region R18 areillustrated in FIG. 3 , the microscope 11 sequentially moves the stageto image all the divided regions corresponding to the observation targetA10 by the high-resolution imaging unit to generate the high-resolutionimages corresponding to the respective divided regions.

When the stage is moved, the glass slide G10 may move on the stage. Whenthe glass slide G10 moves, an unimaged region of the observation targetA10 may occur. As illustrated in FIG. 3 , the microscope 11 captures animage by the high-resolution imaging unit such that the adjacent dividedregions partially overlap each other, and thus, it is possible toprevent the occurrence of an unimaged region even when the glass slideG10 moves.

Note that the low-resolution imaging unit and the high-resolutionimaging unit described above may be different optical systems or may bethe same optical system. In a case of the same optical system, themicroscope 11 changes the resolution according to the imaging target.Furthermore, in the above description, an example is described in whichthe imaging region is changed by moving the stage, but the imagingregion may be changed by the microscope 11 moving an optical system(high-resolution imaging unit or the like). The imaging element providedin the high-resolution imaging unit may be a two-dimensional imagingelement (area sensor) or a one-dimensional imaging element (linesensor). The light from the observation target may be condensed andimaged using the objective lens, or may be dispersed and imaged for eachwavelength using the spectroscopic optical system. In addition, FIG. 3illustrates an example in which the microscope 11 captures an image fromthe central portion of the observation target A10. However, themicroscope 11 may image the observation target A10 in an order differentfrom the imaging order illustrated in FIG. 3 . For example, themicroscope 11 may capture an image from the outer peripheral portion ofthe observation target A10. Furthermore, in the above description, anexample is described in which only the region where the observationtarget A10 exists is imaged by the high-resolution imaging unit.However, since there is a case where the region where the observationtarget A10 exists cannot be accurately extracted, the microscope 11 maydivide the entire region of the imaging region R10 or the glass slideG10 illustrated in FIG. 2 and image the divided regions by thehigh-resolution imaging unit. Note that any method may be used as amethod of capturing a high-resolution image. The divided region may beimaged to acquire a high-resolution image while repeating stop andmovement of the stage, or the divided region may be imaged to acquire ahigh-resolution image on the strip while moving the stage at apredetermined speed.

Subsequently, each high-resolution image generated by the microscope 11is divided into predetermined sizes. As a result, a partial image(hereinafter, it is referred to as a tile image.) is generated from thehigh-resolution image. This point will be described with reference toFIG. 4 . FIG. 4 is a diagram for describing a process of generating apartial image (tile image). FIG. 4 illustrates the high-resolution imageI11 corresponding to the region R11 illustrated in FIG. 3 . Note that,in the following description, it is assumed that a partial image isgenerated from a high-resolution image by the server 12. However, thepartial image may be generated by a device (for example, an informationprocessing device mounted inside the microscope 11, or the like.) otherthan the server 12.

In the example illustrated in FIG. 4 , the server 12 generates 100 tileimages T11, T12, . . . , by dividing one high-resolution image I11. Forexample, in a case where the resolution of the high-resolution image I11is 2560×2560 [pixel], the server 12 generates 100 tile images T11, T12,. . . , each having a resolution of 256×256 [pixel], from thehigh-resolution image I11. Similarly, the server 12 generates tileimages by dividing other high-resolution images by the similar size.

Note that, in the example of FIG. 4 , regions R111, R112, R113, and R114are regions overlapping with adjacent high-resolution images (notillustrated in FIG. 4 ). The server 12 performs positioning of theoverlapping regions by a technique such as template matching to performa stitching process on the adjacent high-resolution images. In thiscase, the server 12 may generate the tile images by dividing thehigh-resolution image after the stitching process. Alternatively, theserver 12 may generate the tile images of regions other than the regionsR111, R112, R113, and R114 before the stitching process, and generatethe tile images of the regions R111, R112, R113, and R114 after thestitching process.

In this manner, the server 12 generates the tile image that is theminimum unit of the captured image of the observation target A10. Then,the server 12 sequentially combines the tile images of the minimum unitto generate the tile images having different hierarchies. Specifically,the server 12 generates one tile image by combining a predeterminednumber of adjacent tile images. This point will be described withreference to FIGS. 5 and 6 . FIGS. 5 and 6 are diagrams for explaining apathological image according to the first embodiment.

The upper part of FIG. 5 illustrates a tile image group of a minimumunit generated from each high-resolution image by the server 12. In theexample in the upper part of FIG. 5 , the server 12 generates one tileimage T110 by combining four tile images T111, T112, T211, and T212adjacent to each other among the tile images. For example, in a casewhere the resolution of each of the tile images T111, T112, T211, andT212 is 256×256, the server 12 generates the tile image T110 having aresolution of 256×256. Similarly, the server 12 generates a tile imageT120 by combining four tile images T113, T114, T213, 1214 adjacent toeach other. In this manner, the server 12 generates the tile imagesobtained by combining a predetermined number of tile images of theminimum unit.

Furthermore, the server 12 generates a tile image obtained by furthercombining tile images adjacent to each other among the tile imagesobtained by combining the tile images of the minimum unit. In theexample of FIG. 5 , the server 12 generates one tile image T100 bycombining four tile images T110, T120, T210, and T220 adjacent to eachother. For example, in a case where the resolution of each of the tileimages T110, T120, T210, and T220 is 256×256, the server 12 generatesthe tile image T100 having a resolution of 256×256. For example, theserver 12 generates an image having a resolution of 256×256 byperforming 4-pixel averaging process, a weighting filtering process(processing of reflecting close pixels more strongly than far pixels), a½ thinning processing, or the like from a tile image having a resolutionof 512×512 obtained by combining four tile images adjacent to eachother.

By repeating such a combining process, the server 12 finally generatesone tile image having a resolution similar to the resolution of the tileimage of the minimum unit. For example, as in the above example, in acase where the resolution of the tile image of the minimum unit is256×256, the server 12 generates one tile image T1 having a resolutionof 256×256 finally by repeating the above-described combining process.

FIG. 6 schematically illustrates the tile image illustrated in FIG. 5 .In the example illustrated in FIG. 6 , the lowermost layer tile imagegroup is tile images of a minimum unit generated by the server 12.Furthermore, the tile image group in the second hierarchy from thebottom is tile images after the lowermost tile image group is combined.Then, the uppermost tile image T1 indicates one tile image to be finallygenerated. In this way, the server 12 generates tile image groups havinga hierarchy such as a pyramid structure illustrated in FIG. 6 as thepathological image.

Note that a region D illustrated in FIG. 5 is an example of a regiondisplayed on a display screen of display device 14 or the like. Forexample, it is assumed that the resolution displayable by the displaydevice corresponds to vertically three tile images and horizontally fourtile images. In this case, as in a region D illustrated in FIG. 5 , thelevel of detail of the observation target A10 displayed on the displaydevice varies depending on the hierarchy to which the tile image to bedisplayed belongs. For example, in a case where the lowermost layer tileimage is used, a narrow region of the observation target A10 isdisplayed in detail. In addition, the wider region of the observationtarget A10 is coarsely displayed as the upper layer tile image is used.

The server 12 stores the tile images of the respective hierarchies asillustrated in FIG. 6 in a storage unit (not illustrated). For example,the server 12 stores each tile image together with tile identificationinformation (an example of partial image information) that can uniquelyidentify each tile image. In this case, in a case of receiving anacquisition request for the tile image including tile identificationinformation from another device (for example, the display control device13), the server 12 transmits a tile image corresponding to the tileidentification information to the another device. Furthermore, forexample, the server 12 may store each tile image together with hierarchyidentification information for identifying each hierarchy and tileidentification information uniquely identifiable in the same hierarchy.In this case, in a case of receiving the acquisition request for thetile image including the hierarchy identification information and thetile identification information from another device, the server 12transmits the tile image corresponding to the tile identificationinformation among the tile images belonging to the hierarchycorresponding to the hierarchy identification information to the anotherdevice.

Note that the server 12 may store the tile images of the respectivehierarchies as illustrated in FIG. 6 in a storage device other than theserver 12. For example, the server 12 may store tile images of eachhierarchy in a cloud server or the like. Furthermore, the process ofgenerating tile images illustrated in FIGS. 5 and 6 may be performed bya cloud server or the like.

Furthermore, the server 12 may not store the tile images of all thehierarchies. For example, the server 12 may store only the tile imagesof the lowermost layer, may store only the tile images of the lowermostlayer and the tile images of the uppermost layer, or may store only thetile images of a predetermined hierarchy (for example, odd-numberedhierarchies, even-numbered hierarchies, and the like.). At this time, ina case where the tile images of an unstored hierarchy is requested fromanother device, the server 12 generates a tile image requested fromanother device by dynamically combining the stored tile images. In thismanner, the server 12 can prevent the storage capacity from beingoverloaded by thinning out the tile images to be stored.

Furthermore, although the imaging conditions are not mentioned in theabove example, the server 12 may store the tile images of the respectivehierarchies as illustrated in FIG. 6 for each imaging condition. Anexample of the imaging condition includes a focal length with respect toa subject (such as the observation target A10). For example, themicroscope 11 may capture images of the same subject while changing thefocal length. In this case, the server 12 may store the tile images ofrespective hierarchies as illustrated in FIG. 6 for each focal length.Note that the reason for changing the focal length is that, since theobservation target A10 is translucent depending on the observationtarget A10, there are a focal length suitable for imaging the surface ofthe observation target A10 and a focal length suitable for imaging theinside of the observation target A10. In other words, the microscope 11can generate a pathological image obtained by imaging the surface of theobservation target A10 and a pathological image obtained by imaging theinside of the observation target A10 by changing the focal length.

In addition, another example of the imaging condition includes astaining condition for the observation target A10. Specifically, in thepathological diagnosis, a specific portion (for example, a cell nucleusor the like) of the observation target A10 may be stained using afluorescent reagent. The fluorescent reagent is, for example, asubstance that is excited and emits light when irradiated with light ofa specific wavelength. Then, different luminescent substances may bestained for the same observation target A10. In this case, the server 12may store tile images of respective hierarchies as illustrated in FIG. 6for each dyed luminescent substance.

Furthermore, the number and resolution of the tile images describedabove are merely examples, and can be appropriately changed depending onthe system. For example, the number of tile images combined by theserver 12 is not limited to four. For example, the server 12 may repeata process of combining 3×3=9 tile images. In the above example, theresolution of the tile image is 256×256, but the resolution of the tileimage may be other than 256×256.

The display control device 13 extracts a desired tile image from thetile image group having the hierarchical structure according to an inputoperation of the user via the display control device 13 using softwareof a system capable of handling the tile image group having thehierarchical structure described above to output the extracted tileimage to the display device 14. Specifically, the display device 14displays an image of an arbitrary portion selected by the user amongimages of aby resolution selected by the user. By such a process, theuser can obtain a feeling as if the user is observing the observationtarget while changing the observation magnification. That is, thedisplay control device 13 functions as a virtual microscope. The virtualobservation magnification here actually corresponds to the resolution.

[2-2. Browsing History Information]

Next, the browsing history information about the pathological imagestored in the server 12 will be described with reference to FIG. 7 .FIG. 7 is a diagram illustrating an example of a browsing mode by aviewer of a pathological image. In the example illustrated in FIG. 7 ,it is assumed that a viewer such as a pathologist has browsed theregions D1, D2, D3, . . . , and D7 in this order in a pathological imageI10. In this case, the display control device 13 first acquires thepathological image corresponding to the region D1 from the server 12according to the browsing operation by the viewer. In response to arequest from the display control device 13, the server 12 acquires oneor more tile images forming the pathological image corresponding to theregion D1 from the storage unit to transmit the acquired one or moretile images to the display control device 13. Then, the display controldevice 13 displays the pathological image formed from the one or moretile images acquired from the server 12 on the display device 14. Forexample, in a case where there is a plurality of tile images, thedisplay control device 13 displays the plurality of tile images side byside. Similarly, each time the viewer performs the operation of changingthe display region, the display control device 13 acquires thepathological image corresponding to the region to be displayed (regionsD2, D3, . . . , D7, etc.) from the server 12 and displays the acquiredpathological image on the display device 14.

In the example of FIG. 7 , since the viewer first browses the relativelywide region D1 and there is no region to be carefully observed in theregion D1, the viewer moves the browsing region to the region D2. Then,since there is a region desired to be carefully observed in the regionD2, the viewer browses the region D3 by enlarging a partial region ofthe region D2. Then, the viewer further moves the browsing region to theregion D4 which is a partial region of the region D2. Then, since thereis a region desired to be observed more carefully in the region D4, theviewer browses the region D5 by enlarging a partial region of the regionD4. In this manner, the viewer is also viewing the regions D6 and D7.For example, each of the pathological images corresponding to theregions D1, D2, and D7 is a display image with a 1.25-foldmagnification, each of the pathological images corresponding to theregions D3 and D4 is a display image with a 20-fold magnification, andeach of the pathological images corresponding to the regions D5 and D6is a display image with a 40-fold magnification. The display controldevice 13 acquires and displays the tile image of the hierarchycorresponding to each magnifications in the tile image group of thehierarchical structure stored in the server 12. For example, thehierarchy of the tile images corresponding to the regions D1 and D2 ishigher (that is, a hierarchy close to the tile image T1 illustrated inFIG. 6 ) than the hierarchy of the tile images corresponding to theregion D3.

While the pathological image is browsed as described above, the displaycontrol device 13 acquires the browsing information at a predeterminedsampling cycle. Specifically, the display control device 13 acquires thecenter coordinates and the display magnification of the browsedpathological image at each predetermined timing, and stores the acquiredbrowsing information in the storage unit of the server 12.

This point will be described with reference to FIG. 8 . FIG. 8 is adiagram illustrating an example of a browsing history storage unit 12 aincluded in the server 12. As illustrated in FIG. 8 , the browsinghistory storage unit 12 a stores information such as “sampling”, “centercoordinates”, “magnification”, and “time”. The “sampling” indicates anorder of timing of storing the browsing information. The “centercoordinates” indicate positional information about the browsedpathological image. In this example, the center coordinates arecoordinates indicated by the center position of the browsed pathologicalimage, and correspond to the coordinates of the coordinate system of thelowermost layer tile image group. The “magnification” indicates adisplay magnification of the browsed pathological image. The “time”indicates an elapsed time from the start of browsing. The example ofFIG. 8 illustrates that the sampling cycle is 30 seconds. That is, thedisplay control device 13 stores the browsing information in thebrowsing history storage unit 12 a every 30 seconds. However, thepresent invention is not limited to this example, and the sampling cyclemay be, for example, 0.1 to 10 seconds, or may be out of this range.

In the example of FIG. 8 , the sampling “1” indicates the browsinginformation about the region D1 illustrated in FIG. 7 , the sampling “2”indicates the browsing information about the region D2, the samplings“3” and “4” indicate the browsing information about the region D3, thesampling “5” indicates the browsing information about the region D4, andthe samplings “6”, “7”, and “8” indicate the browsing information aboutthe region D5. That is, the example of FIG. 8 illustrates that theregion D1 is browsed for about 30 seconds, the region D2 is browsed forabout 30 seconds, the region D3 is browsed for about 60 seconds, theregion D4 is browsed for about 30 seconds, and the region D5 is browsedfor about 90 seconds. In this manner, the browsing time of each regioncan be extracted from the browsing history information.

Furthermore, the number of times each region has been browsed can beextracted from the browsing history information. For example, it isassumed that the number of times of display of each pixel of thedisplayed pathological image is increased by one each time a displayregion changing operation (for example, an operation of moving thedisplay region and an operation of changing the display size) isperformed. For example, in the example illustrated in FIG. 7 , in a casewhere the region D1 is first displayed, the number of times of displayof each pixel included in the region D1 is one. Next, in a case wherethe region D2 is displayed, the number of times of display of each pixelincluded in both the region D1 and the region D2 is two, and the numberof times of display of each pixel included in the region D2 and notincluded in the region D1 is one. Since the display region can beidentified by referring to the center coordinates and the magnificationof the browsing history storage unit 12 a, the number of times eachpixel (that can also be referred to as each coordinates) of thepathological image is displayed can be extracted by analyzing thebrowsing history information stored in the browsing history storage unit12 a.

In a case where the operation of changing the display position is notperformed by the viewer for a predetermined time (for example, 5minutes), the display control device 13 may suspend the storage processof the browsing information. Furthermore, in the above example, anexample is described in which the browsed pathological image is storedas the browsing information by the center coordinates and themagnification, but the present invention is not limited to this example,and the browsing information may be any information as long as it canidentify the region of the browsed pathological image. For example, thedisplay control device 13 may store, as the browsing information aboutthe pathological image, tile identification information for identifyingthe tile image corresponding to the browsed pathological image orinformation indicating the position of the tile image corresponding tothe browsed pathological image. Furthermore, although not illustrated inFIG. 8 , information for identifying a patient, a medical record, andthe like is stored in the browsing history storage unit 12 a. That is,the browsing history storage unit 12 a illustrated in FIG. 8 stores thebrowsing information, the patient, the medical record, and the like inassociation with each other.

[2-3. Diagnostic Information]

Next, diagnostic information stored in a medical information system 30will be described with reference to FIGS. 9A to 9C. FIGS. 9A to 9C arediagrams illustrating a diagnostic information storage unit included inthe medical information system 30. FIGS. 9A to 9C illustrate examples inwhich diagnostic information is stored in different tables forrespective organs to be examined. For example, FIG. 9A illustrates anexample of a table storing diagnostic information related to a breastcancer examination, FIG. 9B illustrates an example of a table storingdiagnostic information related to a lung cancer examination, and FIG. 9Cillustrates an example of a table storing diagnostic information relatedto a large intestine examination.

A diagnostic information storage unit 30A illustrated in FIG. 9A storesinformation such as a “patient ID”, a “pathological image”, a“diagnostic result”, a “grade”, a “tissue type”, a “genetic testing”, an“ultrasonic testing”, and a “medication”. The “patient ID” indicatesidentification information for identifying a patient. The “pathologicalimage” indicates a pathological image stored by the pathologist at thetime of diagnosis. In the “pathological image”, positional information(center coordinates, magnification, and the like) indicating an imageregion to be saved with respect to the entire image may be storedinstead of the image itself. The “diagnosis result” is a diagnosisresult by a pathologist, and indicates, for example, the presence orabsence of a lesion site and the type of lesion site. The “grade”indicates a degree of progression of the diseased site. The “tissuetype” indicates a type of diseased site. The “genetic testing” refers toa result of the genetic testing. The “ultrasonic testing” indicates aresult of the ultrasonic testing. The medication indicates informationabout dosing to the patient.

A diagnostic information storage unit 30B illustrated in FIG. 9B storesinformation related to a “CT testing” performed for lung cancerexamination instead of the “ultrasonic testing” stored in the diagnosticinformation storage unit 30A illustrated in FIG. 9A. A diagnosticinformation storage unit 30C illustrated in FIG. 9C stores informationrelated to an “endoscopic testing” performed for the large intestineexamination instead of the “ultrasonic testing” stored in the diagnosticinformation storage unit 30A illustrated in FIG. 9A.

In the examples of FIGS. 9A to 9C, in a case where the “normal” isstored in the “diagnosis result”, it indicates that the result of thepathological diagnosis is negative, and when information other than the“normal” is stored in the “diagnosis result”, it indicates that theresult of the pathological diagnosis is positive. Note that, in FIGS. 9Ato 9C, the case of storing the patient ID in association with respectiveitems (pathological image, diagnostic result, grade, tissue type,genetic testing, ultrasonic testing, medication) is described. However,it is sufficient to store information related to diagnosis and testingin association with the patient ID, and not all the items are necessary.

3. Image Processing Device According to Present Embodiment

Next, the image processing device 100 according to the presentembodiment will be described. FIG. 10 is a diagram illustrating anexample of the image processing device according to the presentembodiment. As illustrated in FIG. 10 , the image processing device 100includes a communication unit 110, an input unit 120, a display unit130, a storage unit 140, and a control unit 150.

The communication unit 110 is realized by, for example, a networkinterface card (NIC) or the like. The communication unit 110 isconnected to a network (not illustrated) in a wired or wireless manner,to transmit and receives information to and from the pathology system 10and the like via the network. The control unit 150 described latertransmits and receives information to and from these devices via thecommunication unit 110.

The input unit 120 is an input device that inputs various types ofinformation to the image processing device 100. An input unit 111corresponds to a keyboard, a mouse, a touch panel, or the like.

The display unit 130 is a display device that displays informationoutput from the control unit 150. The display unit 130 corresponds to aliquid crystal display, an organic electro luminescence (EL) display, atouch panel, or the like.

The storage unit 140 includes a pathological image data base (DB) 141and a feature amount table 142. For example, the storage unit 140 isrealized by a semiconductor memory device such as a random access memory(RAM) and a flash memory, or a storage device such as a hard disk and anoptical disk.

The pathological image DB 141 is a database that stores a plurality ofpathological images. FIG. 11 is a diagram illustrating an example of adata structure of the pathological image DB. As illustrated in FIG. 11 ,the pathological image DB 141 includes a “patient ID” and a“pathological image”. The patient ID is information for uniquelyidentifying a patient. The pathological image indicates a pathologicalimage stored by the pathologist at the time of diagnosis. Thepathological image is transmitted from the server 12. In addition to thepatient ID and the pathological image, the pathological image DB 141 mayhold information such as the “diagnosis result”, the “grade”, the“tissue type”, the “genetic testing”, the “ultrasonic testing”, and the“medication” described in FIGS. 9A to 9C.

The feature amount table 142 is a table that holds data of featureamounts of partial regions corresponding to cell nuclei and cellmembranes extracted from the pathological image. FIG. 12 is a diagramillustrating an example of a data structure of the feature amount table.As illustrated in FIG. 12 , the feature amount table 142 associates aregion ID, coordinates, and a feature amount. The region ID isinformation for uniquely identifying a partial region. The coordinatesindicate the coordinates (position) of the partial region.

The feature amount is obtained by quantifying characteristics of variouspatterns including a tissue morphology and a state existing in apathological image calculated from a partial region. For example, thefeature amount corresponds to a feature amount output from a neuralnetwork (NN) such as a convolutional neural network (CNN). In addition,the feature amount corresponds to a cell nucleus or a color feature(Luminance, saturation, wavelength, spectrum, and the like), a shapefeature (circularity, circumferential length), a density, a distancefrom a specific morphology, a local feature amount, a structureextraction process (nucleus detection and the like), informationobtained by aggregating them (cell density, orientation, and the like),and the like of a cell nucleus. Here, respective feature amounts areindicated by feature amounts f₁ to f₁₀. Note that the feature amount mayfurther include a feature amount f_(n) other than the feature amounts f₁to f₁₀.

The description returns to FIG. 10 . The control unit 150 includes anacquisition unit 151, an analysis unit 152, a display control unit 153,a generation unit 154, and an image processing unit 155. The controlunit 150 is implemented by, for example, a central processing unit (CPU)or a micro processing unit (MPU) executing a program (an example of animage processing program) stored in the image processing device 100using a random access memory (RAM) or the like as a work area.Furthermore, the control unit 150 is implemented by, for example, anintegrated circuit such as an application specific integrated circuit(ASIC) or a field programmable gate array (FPGA).

The acquisition unit 151 is a processing unit that transmits anacquisition request for the pathological image to the server 12 andacquires the pathological image from the server 12. The acquisition unit151 registers the acquired pathological image in the pathological imageDB 141. The user may operate the input unit 120 to instruct theacquisition unit 151 to acquire the pathological image to be acquired.In this case, the acquisition unit 151 transmits an acquisition requestfor the instructed pathological image to the server 12 to acquire theinstructed pathological image.

The pathological image acquired by the acquisition unit 151 correspondsto a whole slide imaging (WSI). Annotation data indicating part of thepathological image may be attached to the pathological image. Theannotation data indicates a tumor region or the like indicated by apathologist or a researcher. The number of WSIs is not limited to one,and a plurality of WSIs such as serial sections may be included. Inaddition to the patient ID, information such as the “diagnosis result”,the “grade”, the “tissue type”, the “genetic testing”, the “ultrasonictesting”, and “medication” described in FIGS. 9A to 9C may be attachedto the pathological image.

The analysis unit 152 is a processing unit that analyzes thepathological image stored in the pathological image DB 141 andcalculates a feature amount. The user may operate the input unit 120 todesignate the pathological image to be analyzed.

The analysis unit 152 acquires a pathological image designated by theuser from the pathological image DB 141, and extracts a plurality ofpartial regions (patterns) from the pathological image by performingsegmentation on the acquired pathological image. The plurality ofpartial regions includes individual cells, cell organs (cell nucleus,cell membrane, etc.), and cell morphology by a cell or cell organaggregation. In addition, the partial region may be a regioncorresponding to a specific feature possessed in a case where the cellmorphology is normal or in a case of a specific disease. Here, thesegmentation is a technique of assigning a label of an object of a sitein units of pixels from an image. For example, a learned model isgenerated by causing a convolution neural network to learn an image dataset having a correct answer label, and an image (pathological image)desired to be processed is input to the learned model, whereby a labelimage to which a label of an object class is allocated in units ofpixels can be obtained as an output, and a partial region can beextracted for each pixel by referring to the label.

FIG. 13 is a diagram illustrating an example of a partial regionextracted from a pathological image. As illustrated in FIG. 13 , aplurality of partial regions “P” is extracted from the pathologicalimage Ima1. In the following description, in a case where a plurality ofpartial regions is not particularly distinguished, they are simplyreferred to as partial regions. The analysis unit 152 assigns a regionID to a partial region and identifies coordinates of the partial region.The analysis unit 152 registers the coordinates of the partial region inthe feature amount table 142 in association with the region ID.

Subsequently, the analysis unit 152 calculates a feature amount from thepartial region. For example, the analysis unit 152 calculates a featureamount by inputting the image of the partial region to the CNN. Inaddition, the analysis unit 152 calculates a color feature (luminancevalue, dyeing intensity, etc.), a shape feature (circularity,circumferential length, etc.), a density, a distance from a specificmorphology, and a local feature amount based on the image of the partialregion. Any conventional technique may be used for the process in whichthe analysis unit 152 calculates the color feature, the shape feature,the density, the distance from the specific morphology, and the localfeature amount. The analysis unit 152 registers the feature amounts (forexample, the feature amounts f₁ to f₁₀) of the partial region in thefeature amount table 142 in association with the region ID.

The analysis unit 152 may perform the above process after receiving aninstruction for the pathological image to be analyzed from the user, ormay calculate the feature amount of the partial region from a result ofanalyzing the entire pathological image in advance. Furthermore, thepathology system 10 may analyze the entire pathological image, and theanalysis result by the pathology system 10 may be attached to thepathological image, and the analysis unit 152 may calculate the featureamount of the partial region using the analysis result by the pathologysystem 10.

The display control unit 153 is a processing unit that causes thedisplay unit 130 to display screen information about a pathologicalimage indicating the partial region (various patterns including thetissue morphology) extracted by the analysis unit 152 and receivesdesignation of the partial region. For example, the display control unit153 acquires the coordinates of each partial region from the featureamount table 142 and reflects the coordinates in the screen information.

FIG. 14 is a diagram for describing a process by the display controlunit. As illustrated in FIG. 14 , the display control unit 153 displaysthe screen information Dis1 on the display unit 130 such that a partialregion can be designated. The user operates the input unit 120 todesignate some partial regions from the plurality of partial regions anddesignate a category. For example, it is assumed that partial regionsP_(A1), P_(A2), P_(A3), and P_(A4) are selected as the first category.It is assumed that the partial regions P_(B), P_(B2), and P_(B3) areselected as the second category. It is assumed that the partial regionsP_(C1) and P_(C2) are selected as the third category. In the followingdescription, the partial regions P_(A1), P_(A2), P_(A3), and P_(A4) areappropriately collectively referred to as a partial region “P_(A)”. Thepartial regions P_(B), P_(B2), and P_(B3) are appropriately collectivelyreferred to as a partial region “P_(B)”. The partial regions P_(C1) andP_(C2) are appropriately collectively referred to as a partial region“P_(C)”. The display control unit 153 may display partial regionsbelonging to the same category in the same color.

In the example illustrated in FIG. 14 , the process in which the displaycontrol unit 153 receives designation of a partial region is not limitedto the above process. For example, in a case where the user designatesone partial region, the display control unit 153 may automaticallyselect another partial region similar to the shape of the designatedpartial region and determine the another partial region as a partialregion belonging to the same category.

In the example illustrated in FIG. 14 , the case of selecting thepartial region extracted by segmentation is described, but the regiondesignated by the user may be a free region or a geometric region drawnby the user. The display control unit 153 may handle an annotationregion such as a tumor region designated in advance by a pathologist ora researcher as a designated partial region. In addition, the displaycontrol unit 153 may use an extractor for extracting a specific tissueto set a partial region of the tissue extracted by the extractor as adesignated partial region.

The display control unit 153 outputs the region ID of the designatedpartial region and the information about the category of the partialregion to the generation unit 154. In the following description, it isassumed that a region ID of a designated partial region is appropriatelyreferred to as a “designated region ID”, and information about acategory designated by the user is associated with the designated regionID. Note that the display control unit 153 outputs the first to fourthauxiliary information generated by the generation unit 154 to bedescribed later to the display unit 130 to display it on the displayunit 130.

The generation unit 154 is a processing unit that acquires the featureamount corresponding to the designated region ID from the feature amounttable 142 and generates auxiliary information about the feature amountof the pathological image. The auxiliary information includesinformation that enables identification of a feature amount importantfor expressing a feature of a region desired to be classified orextracted, a distribution of feature amounts, and the like. For example,the generation unit 154 generates the first to fourth auxiliaryinformation as the auxiliary information.

The process in which the generation unit 154 generates the “firstauxiliary information” will be described. The generation unit 154calculates a contribution rate (or importance) when classifying orextracting a partial region of the designated region ID for eachcategory with respect to a plurality of feature amounts (for example,the feature amounts f₁ to f₁₀) calculated from the pathological image,and generates first auxiliary information.

As illustrated in FIG. 14 , in a case where the partial region P_(A) ofthe first category, the partial region P_(B) of the second category, andthe partial region P_(C) of the third category are designated, thegeneration unit 154 calculates contribution rates for classifying thepartial regions P_(A), P_(B), and P_(C) based on factor analysis,prediction analysis, and the like. For example, as a result of thefactor analysis, in a case where the contribution rate of the featureamount f₂ among the feature amounts f₁ to f₁₀ increases, it means thatit is appropriate to place a weight on the feature amount f₂ whenclassifying the partial regions P_(A), P_(B), and P_(C). The generationunit 154 generates the first auxiliary information illustrated in FIG.15 .

FIG. 15 is a diagram illustrating an example of the first auxiliaryinformation. As illustrated in FIG. 15 , in the first auxiliaryinformation, the feature amounts f₁ to f₁₀ are associated with thecontribution rates. In the example illustrated in FIG. 15 , since thecontribution rates of the feature amounts f₂, f₆, and f₉ among thefeature amounts f₁ to f₁₀ are large, it means that it is appropriate touse the feature amounts f₂, f₆, and f₉ in a case of classifying thepartial regions P_(A), P_(B), and P_(C). The generation unit 154 outputsthe first auxiliary information to the display control unit 153 torequest the display of the first auxiliary information. The displaycontrol unit 153 displays the first auxiliary information on the displayunit 130. In a case of displaying the first auxiliary information, thedisplay control unit 153 may sort and display the respective featureamounts according to the magnitude of the contribution rate.

The process in which the generation unit 154 generates the “secondauxiliary information” will be described. The generation unit 154compares each feature amount corresponding to the designated region IDwith a threshold value set in advance for each feature amount, andperforms the process for identifying a feature amount equal to orgreater than the threshold value for each category, thereby generatingsecond auxiliary information.

FIG. 16 is a diagram illustrating an example of the second auxiliaryinformation. The generation unit 154 compares the feature amounts f₁ tof₁₀ of the designated region ID corresponding to the first category withthe threshold values Th₁ to Th₁₀ of the respective feature amounts. Forexample, in a case where the feature amount f₁ is equal to or more thanthe threshold value Th₁, the feature amount f₃ is equal to or more thanthe threshold value Th₃, the feature amount f₆ is equal to or more thanthe threshold value Th₆, and the feature amount f₉ is equal to or morethan the threshold value Th₉, the generation unit 154 sets the featureamounts f₁, f₃, f₆, and f₉ as the feature amounts representing thecharacteristics of the first category.

The generation unit 154 compares the feature amounts f₁ to f₁₀ of thedesignated region ID corresponding to the second category with thethreshold values Th₁ to Th₁₀ of the respective feature amounts. Forexample, in a case where the feature amount f₁ is equal to or more thanthe threshold value Th₁ and the feature amount f₃ is equal to or morethan the threshold value Th₃, the generation unit 154 sets the featureamounts f₁ and f₃ as the feature amounts representing thecharacteristics of the second category.

The generation unit 154 compares the feature amounts f₁ to f₁₀ of thedesignated region ID corresponding to the third category with thethreshold values Th₁ to Th₁₀ of the respective feature amounts. Forexample, in a case where the feature amount f₅ is equal to or more thanthe threshold value Th₅, the feature amount f₃ is equal to or more thanthe threshold value Th₃, and the feature amount f₂ is equal to or morethan the threshold value Th₂, the generation unit 154 sets the featureamounts f₅, f₃, and f₂ as the feature amounts representing thecharacteristics of the third category.

The generation unit 154 performs the above process to generate thesecond auxiliary information illustrated in FIG. 16 . In the exampleillustrated in FIG. 16 , in a case where a partial region of the firstcategory is extracted, it means that the feature amounts f₁, f₃, f₆, andf₉ are suitable. In a case where a partial region of the second categoryis extracted, it means that the feature amounts f₁ and f₃ are suitable.In a case where a partial region of the third category is extracted,this means that the feature amounts f₅, f₃, and f₂ are suitable.

The generation unit 154 outputs the second auxiliary information to thedisplay control unit 153 to request the display of the second auxiliaryinformation. The display control unit 153 displays the second auxiliaryinformation on the display unit 130. Note that the display control unit153 may display the second auxiliary information on the display unit 130in the form of a table illustrated in FIG. 17 .

FIG. 17 is a diagram illustrating another display example of the secondauxiliary information. In the row of the feature amounts indicating thecharacteristics of the first category of the display example illustratedin FIG. 17 , the feature amounts f₁, f₃, f₆, and f₉ are indicated bycircles, indicating that the feature amounts f₁, f₃, f₆, and f₉ aresuitable. In the row of the feature amounts indicating thecharacteristics of the second category, the feature amounts f₁, f₃, andf₅ are indicated by circles, indicating that the feature amounts f₁, f₃,and f₅ are suitable. In the row of the feature amounts indicating thecharacteristics of the third category, the feature amounts f₃, and f₂are indicated by circles, indicating that the feature amounts f₅, f₃,and f₂ are suitable. As compared with the display of FIG. 16 , asuitable feature amount and an unsuitable feature amount can be easilygrasped in FIG. 17 .

The process in which the generation unit 154 generates the “thirdauxiliary information” will be described. The generation unit 154generates third auxiliary information in which the distribution ofrespective partial regions is disposed in a feature space having thefirst feature amount f_(i) and the second feature amount f_(j) as axesamong the feature amounts f₁ to f₁₀. The first feature amount f_(i) andthe second feature amount f_(j) may be set in advance, or the featureamounts corresponding to the higher contribution rate may be set as thefirst feature amount f_(i) and the second feature amount f_(j) based onthe contribution rate calculated in the case of generating the firstauxiliary information in FIG. 15 .

FIG. 18 is a diagram illustrating an example of the third auxiliaryinformation. A vertical axis of a feature space Gr1 illustrated in FIG.18 is an axis corresponding to the first feature amount f_(i), and ahorizontal axis is an axis corresponding to the second feature amountf_(j). The generation unit 154 refers to the feature amount table 142,identifies the first feature amount f_(i) and the second feature amountf_(j) of each partial region, and plots a point corresponding to eachpartial region on the feature space Gr1. In addition, the generationunit 154 sets the point of the partial region corresponding to thedesignated region ID among the points corresponding to the respectivepartial regions to be identifiable.

For example, in a case where the partial region P_(A1) of the firstcategory designated in FIG. 14 corresponds to the point do1 in FIG. 18 ,the generation unit 154 disposes the point do1 in the first colorindicating that the partial region P_(A1) belongs to the first category.In a case where the partial region P_(A2) of the first categorydesignated in FIG. 14 corresponds to the point dot in FIG. 18 , thepoint do1 is disposed in the first color indicating that the partialregion P_(A2) belongs to the first category.

In a case where the partial region PB1 of the second category designatedin FIG. 14 corresponds to the point do3 in FIG. 18 , the generation unit154 disposes the point do3 in the second color indicating that thepartial region PB1 belongs to the second category. In a case where thepartial region P_(C1) of the third category designated in FIG. 14corresponds to the point do4 in FIG. 18 , the point do4 is disposed inthe third color indicating that the partial region P_(C1) belongs to thethird category. It is assumed that the first color, the second color,and the third color are different colors.

The generation unit 154 outputs the third auxiliary information to thedisplay control unit 153 to request the display of the third auxiliaryinformation. The display control unit 153 displays the third auxiliaryinformation on the display unit 130. Note that, in the multidimensionalfeature amount, the generation unit 154 may calculate thelow-dimensional feature amount obtained using dimension reduction suchas principal component analysis or TSNE, and plot points correspondingto respective partial regions in the feature space of thelow-dimensional feature amount.

The process in which the generation unit 154 generates the “fourthauxiliary information” will be described. Based on the feature amounttable 142, the generation unit 154 generates a histogram of each of thefeature amounts f₁ to f₁₀ as the fourth auxiliary information.

FIG. 19 is a diagram illustrating an example of the fourth auxiliaryinformation. FIG. 19 illustrates histograms h1-1 to h4-1 of therespective feature amounts as an example. The generation unit 154 mayset the frequency of the class value corresponding to the feature amountof the partial region corresponding to the designated region ID to beidentifiable in each histogram.

The histogram h1-1 is a histogram corresponding to the feature amountf₁. In a case where the feature amount f₁ of the partial region P_(A1)of the first category corresponds to the class value cm1, the generationunit 154 sets the color of the frequency corresponding to the classvalue cm1 to the first color. In a case where the feature amount f₁ ofthe partial region P_(B1) of the second category corresponds to theclass value cm2, the generation unit 154 sets the color of the frequencycorresponding to the class value cm2 to the second color. In a casewhere the feature amount f₁ of the partial region P_(C1) of the thirdcategory corresponds to the class value cm3, the generation unit 154sets the color of the frequency corresponding to the class value cm3 tothe third color.

The histogram h2-1 is a histogram corresponding to the feature amountf₂. In a case where the feature amount f₂ of the partial region P_(A1)of the first category corresponds to the class value cm1, the generationunit 154 sets the color of the frequency corresponding to the classvalue cm1 to the first color. In a case where the feature amount f₂ ofthe partial region P_(B1) of the second category corresponds to theclass value cm2, the generation unit 154 sets the color of the frequencycorresponding to the class value cm2 to the second color. In a casewhere the feature amount f₂ of the partial region P_(C1) of the thirdcategory corresponds to the class value cm3, the generation unit 154sets the color of the frequency corresponding to the class value cm3 tothe third color.

The histogram h3-1 is a histogram corresponding to the feature amountf₃. In a case where the feature amount f₃ of the partial region P_(A1)of the first category corresponds to the class value cm1, the generationunit 154 sets the color of the frequency corresponding to the classvalue cm1 to the first color. In a case where the feature amount f₃ ofthe partial region P_(B1) of the second category corresponds to theclass value cm2, the generation unit 154 sets the color of the frequencycorresponding to the class value cm2 to the second color. In a casewhere the feature amount f₃ of the partial region P_(C1) of the thirdcategory corresponds to the class value cm3, the generation unit 154sets the color of the frequency corresponding to the class value cm3 tothe third color.

The histogram h4-1 is a histogram corresponding to the feature amountf₄. In a case where the feature amount f₄ of the partial region P_(A1)of the first category corresponds to the class value cm1, the generationunit 154 sets the color of the frequency corresponding to the classvalue cm1 to the first color. In a case where the feature amount f₄ ofthe partial region P_(B1) of the second category corresponds to theclass value cm2, the generation unit 154 sets the color of the frequencycorresponding to the class value cm2 to the second color. In a casewhere the feature amount f₄ of the partial region P_(C1) of the thirdcategory corresponds to the class value cm3, the generation unit 154sets the color of the frequency corresponding to the class value cm3 tothe third color.

Although not illustrated, the generation unit 154 similarly generateshistograms corresponding to the feature amounts f₅ to f₁₀. Thegeneration unit 154 outputs the fourth auxiliary information to thedisplay control unit 153 to request the display of the fourth auxiliaryinformation. The display control unit 153 displays the fourth auxiliaryinformation on the display unit 130.

Here, the display control device 153 may display all the 1st to 4thauxiliary information on the display unit 130, or may display only partof the auxiliary information. Furthermore, the user may operate theinput unit 120 to designate the auxiliary information to be displayed.In the following description, in a case where the first to fourthauxiliary information is not particularly distinguished, they are simplyreferred to as auxiliary information.

In addition, after referring to the auxiliary information, the user mayoperate an input unit 130, refer to screen information Dis1 illustratedin FIG. 14 again, and select the partial region and the category of thepartial region again. In a case where the partial region and thecategory of the partial region are reselected, the display control unit153 outputs a new designated region ID to the generation unit 154. Thegeneration unit 154 generates new auxiliary information based on the newdesignated region ID, and the display control unit 153 outputs the newauxiliary information to the display unit 130 to display it on thedisplay unit 130. The display control unit 153 and the generation unit154 repeatedly perform the above process each time the user reselectsthe partial region and the category of the partial region.

The description returns to FIG. 10 . The image processing unit 155 is aprocessing unit that performs various types of image processes on apathological image in a case where designation of the pathological imageis received from the user. For example, based on the parameter, theimage processing unit 155 performs the process of classifying a partialregion included in the pathological image according to the featureamount, the process of extracting a partial region having a specificfeature amount, and the like. The parameter is set by the user whorefers to the auxiliary information.

In a case where the image processing unit 155 performs the image processof classifying the partial region included in the pathological imageaccording to the feature amount, the user operates the input unit 120 toset, as parameters, the feature amount (some feature amounts of thefeature amounts f₁ to f₁₀) to be used at the time of classification, theimportance of the feature amount, and the like.

In a case where the image processing unit 155 performs the image processof extracting a partial region having a specific feature amount frompartial regions included in the pathological image, the user operatesthe input unit 120 to set, as parameters, the feature amount (somefeature amounts of the feature amounts f₁ to f₁₀) to be used at the timeof extraction, the threshold value for each feature amount at the timeof extraction, and the like.

Here, an example of a processing result by the image processing unit 155will be described. FIGS. 20, 21 , and 22 are diagrams for explaining anexample of a classification process performed by the image processingunit 155. FIG. 20 will be described. A pathological image Ima1-1 is apathological image before the classification process is performed. Inthe pathological image Ima1-1, the user designates the partial regionP_(A) as the first category, designates the partial region P_(B) as thesecond category, and selects the partial region P_(C) as the thirdcategory. The partial region P_(A) is indicated by the first color. Thepartial region P_(B) is indicated by the second color. The partialregion P_(C) is indicated in the third color.

The image processing unit 155 classifies each partial region included inthe pathological image Ima1-1 into one of the first category, the secondcategory, and the third category based on the parameter set to the user.The classification result is illustrated in a pathological image Ima1-2.In the pathological image Ima1-2, each partial region indicated by thefirst color is a partial region classified into the first category. Eachpartial region indicated by the second color is a partial regionclassified into the second category. Each partial region indicated bythe third color is a partial region classified into the third category.The image processing unit 155 may output the pathological image Ima1-2as the classification result to the display unit 130 to display it onthe display unit 130.

FIG. 21 will be described. FIG. 21 illustrates a case where the imageprocessing unit 155 plots partial regions classified into the firstcategory, the second category, and the third category in the featurespace Gr1 according to the feature amount. The vertical axis of thefeature space Gr1 is an axis corresponding to the first feature amountf_(i), and the horizontal axis is an axis corresponding to the secondfeature amount f_(j). In the example illustrated in FIG. 21 , thepartial regions classified into the first category are located in aregion Ar1. The partial region classified into the second category islocated in a region Art. The partial region classified into the thirdcategory is located in a region Ar3. The image processing unit 155 mayoutput the information about the feature space Gr1 illustrated in FIG.21 to the display unit 130 to display it on the display unit 130.

FIG. 22 will be described. In FIG. 22 , histograms h1-2 to h4-2 of therespective feature amounts are illustrated. The image processing unit155 generates histograms h1-2 to h4-2 by making the distribution of thefeature amounts of the partial regions classified into the firstcategory, the distribution of the feature amounts of the partial regionsclassified into the second category, and the distribution of the featureamounts of the partial regions classified into the third categoryidentifiable with each other.

The histogram h1-2 is a histogram corresponding to the feature amountf₁. In the histogram h1-2, a distribution 41 a is a distribution of thefeature amounts of the partial regions classified into the firstcategory. A distribution 42 a is a distribution of the feature amountsof the partial regions classified into the second category. Adistribution 43 a is a distribution of the feature amounts of thepartial regions classified into the third category.

The histogram h2-2 is a histogram corresponding to the feature amountf₂. In the histogram h2-2, the distribution 41 b is a distribution ofthe feature amounts of the partial regions classified into the firstcategory. The distribution 42 b is a distribution of the feature amountsof the partial regions classified into the second category. Thedistribution 43 b is a distribution of the feature amounts of thepartial regions classified into the third category.

The histogram h3-2 is a histogram corresponding to the feature amountf₃. In the histogram h3-2, the distribution 41 c is a distribution ofthe feature amounts of the partial regions classified into the firstcategory. The distribution 42 c is a distribution of the feature amountsof the partial regions classified into the second category. Thedistribution 43 c is a distribution of the feature amounts of thepartial regions classified into the third category.

The histogram h4-2 is a histogram corresponding to the feature amountf₄. In the histogram h4-2, the distribution 41 d is a distribution ofthe feature amounts of the partial regions classified into the firstcategory. The distribution 42 d is a distribution of the feature amountsof the partial regions classified into the second category. Thedistribution 43 d is a distribution of the feature amounts of thepartial regions classified into the third category.

Although not illustrated, the image processing unit 155 similarlygenerates histograms corresponding to the feature amounts f₅ to f₁₀. Theimage processing unit 155 may output the information about thehistograms h1-2 to h4-2 illustrated in FIG. 22 to the display unit 130to display it on the display unit 130.

4. Processing Procedure

FIG. 23 is a flowchart illustrating a processing procedure of the imageprocessing device 100 according to the present embodiment. Asillustrated in FIG. 23 , the acquisition unit 151 of the imageprocessing device 100 acquires a pathological image (step S101). Theanalysis unit 152 of the image processing device 100 performssegmentation on the pathological image and extracts a partial region(step S102).

The analysis unit 152 calculates a feature amount of each partial region(step S103). The display control unit 153 displays the pathologicalimage indicating the partial region on the display unit 130 (step S104).The display control unit 153 receives designation of the partial region(step S105).

The generation unit 154 of the image processing device 100 generatesauxiliary information (step S106). The display control unit 153 displaysthe auxiliary information on the display unit 130 (step S107).

In a case of receiving a change or an addition of the partial region tobe designated (step S108, Yes), the image processing device 100 advancesthe process to step S105. On the other hand, in a case of not receivingthe change or addition of the partial region to be designated (stepS108, No), the image processing device 100 advances the process to stepS109.

The image processing unit 155 of the image processing device 100receives adjustment of parameters (step S109). The image processing unit155 performs the classification or extraction process based on theadjusted parameters (step S110).

In a case where the re-adjustment of the parameter is received (stepS111, Yes), the image processing device 100 advances the process to stepS109. In a case where re-adjustment of the parameters is not received(step S111, No), the image processing device 100 ends the process.

5. Another Process

The image processing device 100 may generate, as the auxiliaryinformation, information capable of grasping the situation of aplurality of partial regions in the pathological image, such as thesituation of the entire pathological image, and display the auxiliaryinformation.

FIGS. 24 and 25 are diagrams for explaining another process by the imageprocessing device 100. FIG. 24 will be described. The display controlunit 153 of the image processing device 100 displays a pathologicalimage Ima10 divided into a plurality of regions of interest (ROIs). Theuser operates the input unit 120 to designate a plurality of ROIs. Inthe example illustrated in FIG. 24 , a case where ROIs 40 a, 40 b, 40 c,40 d, and 40 e are designated is illustrated. When receiving thedesignation of the ROI, the display control unit 153 displays the screeninformation illustrated in FIG. 25 .

FIG. 25 will be described. The display control unit 153 displays theenlarged ROI images 41 a to 41 e in screen information 45. The image 41a is an enlarged image of the ROI 40 a. The image 41 b is an enlargedimage of the ROI 40 b. The image 41 c is an enlarged image of the ROI 40c. The image 41 d is an enlarged image of the ROI 40 d. The image 41 eis an enlarged image of the ROI 40 e.

The analysis unit 152 of the image processing device 100 extractspartial regions from the ROI 40 a and calculates a feature amount ofeach partial region in the same manner as the above processing. Thegeneration unit 154 of the image processing device 100 generatesauxiliary information 42 a based on the feature amounts of respectivepartial regions of the ROI 40 a and sets the auxiliary information inthe screen information 45. For example, the auxiliary information 42 amay be the third auxiliary information described with reference to FIG.18 , or may be another auxiliary information. The generation unit 154also generates auxiliary information 42 b to 42 e based on the featureamounts of respective partial regions of the ROIs 40 b to 40 e,respectively, and sets the auxiliary information in the screeninformation 45.

The user can grasp the features of the entire pathological image byreferring to the screen information 45, and can use the features forparameter adjustment in a case where the image process is performed.

6. Effects of Image Processing Device According to Present Embodiment

The image processing device 100 according to the present embodimentextracts a plurality of partial regions from a pathological image, andgenerates auxiliary information indicating a feature amount effective ina case of classifying or extracting a partial image with respect to aplurality of feature amounts calculated from the pathological image whendesignation of the partial region is received. In a case of receivingthe setting of the parameter from the user who has referred to theauxiliary information, the image processing device 100 performs theimage process on the pathological image using the received parameter. Asa result, the feature amount obtained by quantifying the appearancefeature of the morphology can be appropriately displayed by theauxiliary information, and adjustment of the parameter of the imageprocess can be facilitated. For example, the “macroscopic and visiblefeature” of a specialist such as a pathologist can be easily associatedwith the “calculated quantitative feature”.

The image processing device 100 calculates a contribution rate whenclassifying the plurality of designated partial regions, and generatesand displays, as auxiliary information, information in which the featureamount and the contribution rate are associated with each other. Byreferring to such auxiliary information, the user can easily grasp whichfeature amount should be emphasized to set the parameter in a case ofclassifying the plurality of partial regions for each category.

The image processing device 100 selects some feature amounts based onthe magnitudes of the plurality of feature amounts calculated from theplurality of designated partial regions, and generates the selectedfeature amount as auxiliary information. By referring to such auxiliaryinformation, the user can easily grasp the feature amount to be used ina case of extracting a partial region having the category same as thatof the designated partial region.

The image processing device 100 performs segmentation on thepathological image and extracts a plurality of partial regions. As aresult, the user can easily designate the region corresponding to thecell morphology included in the pathological image.

The image processing device 100 displays all the partial regionsincluded in the pathological image, and receives selection of aplurality of partial regions among all the partial regions. As a result,the user can easily select the partial region to be used in the creationof the auxiliary information.

The image processing device 100 performs the factor analysis, predictionanalysis, or the like to calculate a contribution rate. As a result, itis possible to calculate the feature amount effective in a case wherethe partial region is appropriately classified for each of thedesignated different categories.

The image processing device 100 generates a feature space correspondingto some feature amounts, and identifies a position in the feature amountspace corresponding to the partial region designation of which wasreceived based on the feature amount of the partial region designationof which was received. As a result, the user can easily grasp theposition in the feature space with respect to the designated partialregion.

The image processing device 100 identifies a feature amount having ahigh contribution rate and generates a feature space of the identifiedfeature amount. As a result, the user can grasp the distribution of thedesignated partial regions in the feature space of the feature amountwith a high contribution rate.

In a case where a plurality of ROIs is designated for the entirepathological image, the image processing device 100 generates auxiliaryinformation based on the feature amount of the partial region includedin each ROI. As a result, it is possible to grasp the features of theentire pathological image and to use the features for parameteradjustment when the image process is performed.

3. Hardware Configuration

The image processing device according to each embodiment described aboveis implemented by, for example, a computer 1000 having a configurationas illustrated in FIG. 26 . Hereinafter, an imaging system 100 accordingto the embodiments will be described as an example. FIG. 26 is ahardware configuration diagram illustrating an example of the computer1000 that implements the functions of the image processing device. Thecomputer 1000 includes a CPU 1100, a RAM 1200, a read only memory (ROM)1300, a hard disk drive (HDD) 1400, a communication interface 1500, andan input/output interface 1600. Respective units of the computer 1000are connected by a bus 1050.

The CPU 1100 operates based on a program stored in the ROM 1300 or theHDD 1400, and controls each unit. For example, the CPU 1100 develops aprogram stored in the ROM 1300 or the HDD 1400 in the RAM 1200, andexecutes processing corresponding to various programs.

The ROM 1300 stores a boot program such as a basic input output system(BIOS) executed by the CPU 1100 when the computer 1000 is activated, aprogram depending on hardware of the computer 1000, and the like.

The HDD 1400 is a computer-readable recording medium thatnon-transiently records programs executed by the CPU 1100, data used bythe programs, and the like. Specifically, the HDD 1400 is a recordingmedium that records an information processing program according to thepresent disclosure which is an example of program data 1450.

The communication interface 1500 is an interface for the computer 1000to be connected to an external network 1550 (for example, the Internet).For example, the CPU 1100 receives data from another device or transmitsdata generated by the CPU 1100 to another device via the communicationinterface 1500.

The input/output interface 1600 is an interface that connects aninput/output device 1650 and the computer 1000. For example, the CPU1100 receives data from an input device such as a keyboard and a mousevia the input/output interface 1600. In addition, the CPU 1100 transmitsdata to an output device such as a display, a speaker, or a printer viathe input/output interface 1600. Furthermore, the input/output interface1600 may function as a media interface that reads a program or the likerecorded in a predetermined recording medium (medium). The medium is,for example, an optical recording medium such as a digital versatiledisc (DVD) or a phase change rewritable disk (PD), a magneto-opticalrecording medium such as a magneto-optical disk (MO), a tape medium, amagnetic recording medium, a semiconductor memory, or the like.

Furthermore, the computer 1000 is connected to a millimeter wave radaror a camera module (corresponding to an image generation unit 107 or thelike) via the input/output interface 1600.

For example, in a case where the computer 1000 functions as the imageprocessing device 100 according to the embodiments, the CPU 1100 of thecomputer 1000 executes the image processing program loaded on the RAM1200 to implement the functions of the acquisition unit 151, theanalysis unit 152, the display control unit 153, the generation unit154, the image processing unit 155, and the like. Further, the HDD 1400stores an image processing program or the like according to the presentdisclosure. The CPU 1100 reads the program data 1450 from the HDD 1400and executes the program data, but as another example, the program maybe acquired from another device via the external network 1550.

4. Conclusion

The image processing device includes a generation unit and an imageprocessing unit. In a case of receiving designation of a plurality ofpartial regions extracted from a pathological image and corresponding toa cell morphology, the generation unit generates auxiliary informationindicating information about a feature amount effective when classifyingor extracting the plurality of partial regions with respect to theplurality of feature amounts calculated from the image. In a case ofreceiving setting information about an adjustment item according to theauxiliary information, the image processing unit performs the imageprocess on the image using the setting information. As a result, thefeature amount obtained by quantifying the appearance feature of themorphology can be appropriately displayed by the auxiliary information,and adjustment of the parameter of the image process can be facilitated.For example, it is possible to easily associated the “feature based onknowledge of a specialist such as a pathologist” with the “calculatedquantitative feature”.

The generation unit calculates a contribution rate when classifying theplurality of designated partial regions, and generates, as the auxiliaryinformation, information in which the feature amount and thecontribution rate are associated with each other. By referring to suchauxiliary information, the user can easily grasp which feature amountshould be emphasized to set the parameter in a case of classifying theplurality of partial regions for each category.

The generation unit selects some feature amounts based on the magnitudesof the plurality of feature amounts calculated from the plurality ofdesignated partial regions, and generates information about the selectedfeature amounts as the auxiliary information. By referring to suchauxiliary information, the user can easily grasp the feature amount tobe used in a case of extracting a partial region having the categorysame as that of the designated partial region.

The image processing device performs segmentation on the image andextracts the plurality of partial regions. As a result, the user caneasily designate the region corresponding to the cell morphologyincluded in the pathological image.

The image processing device further includes a display control unit thatdisplays all partial regions extracted by the analysis unit and receivesdesignation of a plurality of partial regions among all the partialregions. The display control unit further displays the auxiliaryinformation. As a result, the user can easily select the partial regionto be used in the creation of the auxiliary information.

The generation unit performs the factor analysis or the predictionanalysis to calculate the contribution rate. As a result, it is possibleto calculate the feature amount effective in a case where the partialregion is appropriately classified for each of the designated differentcategories.

The generation unit generates a feature space corresponding to somefeature amounts, and identifies a position in the feature amount spacecorresponding to the partial region designation of which was receivedbased on a feature amount of a partial region designation of which wasreceived. As a result, the user can easily grasp the position in thefeature space with respect to the designated partial region.

The generation unit identifies a feature amount having a highcontribution rate and generates a feature space of the identifiedfeature amount. As a result, the user can grasp the distribution of thedesignated partial regions in the feature space of the feature amountwith a high contribution rate.

In a case where a plurality of regions is designated for thepathological image, the generation unit generates auxiliary informationfor each of the plurality of regions. As a result, it is possible tograsp the features of the entire pathological image and to use thefeatures for parameter adjustment when the image process is performed.

REFERENCE SIGNS LIST

-   -   1 DIAGNOSIS SUPPORT SYSTEM    -   10 PATHOLOGY SYSTEM    -   11 MICROSCOPE    -   12 SERVER    -   13 DISPLAY CONTROL DEVICE    -   14 DISPLAY DEVICE    -   100 IMAGE PROCESSING DEVICE    -   110 COMMUNICATION UNIT    -   120 INPUT UNIT    -   130 DISPLAY UNIT    -   140 STORAGE UNIT    -   141 PATHOLOGICAL IMAGE DB    -   142 FEATURE AMOUNT TABLE    -   150 CONTROL UNIT    -   151 ACQUISITION UNIT    -   152 ANALYSIS UNIT    -   153 DISPLAY CONTROL UNIT    -   154 GENERATION UNIT    -   155 IMAGE PROCESSING UNIT

1. An image processing device including: in a case where designation ofa plurality of partial regions corresponding to a cell morphology isreceived, the plurality of partial regions being extracted from apathological image, a generation unit that generates auxiliaryinformation indicating information about a feature amount effective whena plurality of partial regions is classified or extracted with respectto a plurality of feature amounts calculated from the image; and in acase where setting information about an adjustment item according to theauxiliary information is received, an image processing unit thatperforms an image process on the image using the setting information. 2.The image processing device according to claim 1, wherein the generationunit calculates a contribution rate when the plurality of designatedpartial regions is classified, and generates, as the auxiliaryinformation, information in which the feature amount and thecontribution rate are associated with each other.
 3. The imageprocessing device according to claim 1, wherein the generation unitselects some feature amounts based on magnitudes of a plurality offeature amounts calculated from a plurality of designated partialregions, and generates information about the selected feature amounts asthe auxiliary information.
 4. The image processing device according toclaim 1, further including: an analysis unit that performs segmentationon the image and extracts the plurality of partial regions.
 5. The imageprocessing device according to claim 4, further including: a displaycontrol unit that displays all partial regions extracted by the analysisunit and receives designation of a plurality of partial regions amongall the partial regions.
 6. The image processing device according toclaim 5, wherein the display control unit further displays the auxiliaryinformation.
 7. The image processing device according to claim 2,wherein the generation unit performs a factor analysis or a predictionanalysis to calculate the contribution rate.
 8. The image processingdevice according to claim 5, wherein the generation unit generates afeature space corresponding to some feature amounts, and identifies aposition in the feature amount space corresponding to a partial regiondesignation of which was received based on a feature amount of thepartial region designation of which was received.
 9. The imageprocessing device according to claim 6, wherein the generation unitidentifies a feature amount having a high contribution rate andgenerates a feature space of the identified feature amount.
 10. Theimage processing device according to claim 1, wherein in a case where aplurality of regions is designated for the pathological image, thegeneration unit generates auxiliary information for each of theplurality of regions.
 11. An image processing method executed by acomputer, the method including: in a case where designation of aplurality of partial regions corresponding to a cell morphology isreceived, the plurality of partial regions being extracted from apathological image, generating auxiliary information indicatinginformation about a feature amount effective when a plurality of partialregions is classified or extracted with respect to a plurality offeature amounts calculated from the image; and in a case where settinginformation about an adjustment item according to the auxiliaryinformation is received, performing an image process on the image usingthe setting information.
 12. An image processing program for causing acomputer to function as: in a case where designation of a plurality ofpartial regions corresponding to a cell morphology is received, theplurality of partial regions being extracted from a pathological image,a generation unit that generates auxiliary information indicatinginformation about a feature amount effective when a plurality of partialregions is classified or extracted with respect to a plurality offeature amounts calculated from the image; and in a case where settinginformation about an adjustment item according to the auxiliaryinformation is received, an image processing unit that performs an imageprocess on the image using the setting information.
 13. A diagnosissupport system including: a medical image acquisition device andsoftware used for processing a medical image corresponding to an objectimaged by the medical image acquisition device, wherein the softwarecauses an image processing device to in a case where designation of aplurality of partial regions corresponding to a cell morphology isreceived, the plurality of partial regions being extracted from apathological image, generate auxiliary information indicatinginformation about a feature amount effective when a plurality of partialregions is classified or extracted with respect to a plurality offeature amounts calculated from the image; and in a case where settinginformation about an adjustment item according to the auxiliaryinformation is received, perform an image process on the image using thesetting information.